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Abstract 

Background: Health-system policy makers need timely access to synthesised research evidence to inform the policy- 
making process. No efforts to address this need have been evaluated using an experimental quantitative design. We 
developed an evidence service that draws inputs from Health Systems Evidence, which is a database of policy-relevant 
systematic reviews. The reviews have been (a) categorised by topic and type of review; (b) coded by the last year 
searches for studies were conducted and by the countries in which included studies were conducted; (c) rated for 
quality; and (d) linked to available user-friendly summaries, scientific abstracts, and full-text reports. Our goal is to evaluate 
whether a "full-serve" evidence service increases the use of synthesized research evidence by policy analysts and advisors 
in the Ontario Ministry of Health and Long-Term Care (MOHLTC) as compared to a "self-serve" evidence service. 

Methods/design: We will conduct a two-arm randomized controlled trial (RCT), along with a follow-up qualitative 
process study in order to explore the findings in greater depth. For the RCT, all policy analysts and policy advisors (n = 
168) in a single division of the MOHLTC will be invited to participate. Using a stratified randomized design, participants 
will be randomized to receive either the "full-serve" evidence service (database access, monthly e-mail alerts, and full-text 
article availability) or the "self-serve" evidence service (database access only). The trial duration will be ten months (two- 
month baseline period, six-month intervention period, and two month cross-over period). The primary outcome will be 
the mean number of site visits/month/user between baseline and the end of the intervention period. The secondary 
outcome will be participants' intention to use research evidence. For the qualitative study, 15 participants from each trial 
arm (n = 30) will be purposively sampled. One-on-one semi-structured interviews will be conducted by telephone on 
their views about and their experiences with the evidence service they received, how helpful it was in their work why it 
was helpful (or not helpful), what aspects were most and least helpful and why, and recommendations for next steps. 

Discussion: To our knowledge, this will be the first RCT to evaluate the effects of an evidence service specifically 
designed to support health-system policy makers in finding and using research evidence. 

Trial registration: ClinicalTrials.gov: NCTOl 307228 



Background 

Health- system policy makers make important decisions 
every day about the governance, financial, and delivery 
arrangements within which programs, services, and 
drugs are provided and about implementation strategies 
[1]. The nature of their decisions will vary according to 
the setting in which they work {e.g., federal, provincial. 
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or local government) and the role they play (e.g., politi- 
cal staff, policy analyst, senior policy advisor. Assistant 
Deputy Minister, or elected official), among other fac- 
tors. Systematic reviews are increasingly seen as a key 
source of information to inform these decisions [1]. 
Reduced bias and increased precision comprise the main 
advantages of systematic reviews that address questions 
about the effects of interventions [2]. Drawing on a sys- 
tematic review that addresses any question constitutes a 
more efficient use of time for busy policy makers 
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because the research Uterature has already been identi- 
fied, selected, appraised, and synthesised in a systematic 
and transparent way. Additionally, a systematic review 
makes possible more constructive policy debates because 
stakeholders can focus on the synthesis and its local 
applicability rather than on which single study has 
greater credibility [3]. 

In order to make informed decisions, health-system 
policy makers need timely access to systematic reviews 
that can be easily retrieved using terminology that is 
understandable to them and that are presented in ways 
that facilitate rapid scanning for relevance, recency of 
searches for potentially relevant studies, the settings of 
studies included in the review, and quality of the review 
[3,4]. A systematic review of the factors that influence 
the use of research in policy making identified timing/ 
timeliness as one of two factors that increased the pro- 
spects for research use among health-system policy 
makers [3,5]. However, when attempting to retrieve sys- 
tematic reviews in a timely fashion, health-system policy 
makers typically cannot search all of the potential 
sources of systematic reviews. Moreover, policy makers 
typically cannot search most sources of systematic 
reviews, like The Cochrane Library, using terms with 
which they are familiar. The number and searchability 
of existing sources of systematic reviews become parti- 
cularly frustrating when policy makers know there is 
likely to be a review available on a topical issue. More- 
over, search results typically do not highlight the types 
of decision-relevant information that health-system pol- 
icy makers are seeking [3,4]. 

One response to the similar types of issues faced by 
clinical decision makers has been the development of 
evidence services that provide regular email alerts about 
newly identified research products and a searchable 
database of these products [6]. However, no 'full-serve' 
evidence service currently exists to meet the needs of 
health-system policy makers. Existing evidence services 
that include health-system policy makers among their 
target audiences, such as E-watch (http://kuuc.chair.ula- 
val.ca/english/index.php) and CHAIN Canada (http:// 
www.epoc.uottawa.ca/CHAINCanada/), do not focus on 
systematic reviews. Existing evidence services that focus 
on high-quality studies (not just systematic reviews), 
such as Evidence Updates (http://plus.mcmaster.ca/Evi- 
denceUpdates/), do not target health-system policy 
makers [6]. 

To address this gap, we developed a full-serve evi- 
dence service for health-system policy makers. First, we 
developed Health Systems Evidence, which contains 
over 1,400 syntheses about governance, financial, and 
delivery arrangements within health systems and about 
implementation strategies relevant to health systems. By 
syntheses we mean both systematic reviews and two 



types of review-derived products, namely, policy briefs 
and overviews of systematic reviews [7]. A policy brief 
summarises how the findings from a number of sys- 
tematic reviews pertain to a pressing problem, select 
options for addressing the problem, and key implemen- 
tation considerations, whereas an overview provides a 
'map' of all available systematic reviews on a broad 
health-system topic. The reviews have been (a) cate- 
gorised by topic (i.e., by health-system arrangement or 
implementation strategy), type of review {i.e., policy 
brief, overview of reviews, Cochrane systematic review, 
systematic review, or systematic review protocol), and 
type of question addressed {i.e., effectiveness, not effec- 
tiveness, and 'many'); (b) coded by the last year in which 
searches for studies were conducted and by the coun- 
tries in which included studies were conducted; (c) 
rated for quality using the AMSTAR (A MeaSurement 
Tool for the 'Assessment of multiple systematic 
Reviews') instrument [8,9]; and (d) linked to available 
user-friendly summaries, scientific abstracts, and full- 
text reviews that are available free online [10]. 

Second, we identified systematic reviews in Health 
Systems Evidence that are neither available free online 
nor available through subscriptions held by the Ontario 
Ministry of Health and Long-Term Care (MOHLTC) 
and developed a mechanism to reimburse publishers for 
full-text downloads of these reviews. 

Third, we developed the format for monthly email 
alerts, which (in tabular format) identifies new additions 
to Health Systems Evidence and describes the type of 
review, type of question addressed, health-system 
arrangement or implementation strategy addressed, and 
title of the review. A hypertext link for each review 
enables policy makers to view the availability of (and 
links to) user-friendly summaries, scientific abstracts, 
and the full-text review. A hypertext link to the online 
Health Systems Evidence webpage enables policy makers 
to view additional information about these same recent 
database additions, including the last year searched, 
quality rating, the countries in which included studies 
were conducted, and the complete citation. (Electronic 
newsletter width restrictions precluded having all fields 
presented in the monthly email alerts.) 

Our goal is to evaluate whether (and how and why) a 
full-serve evidence service increases the use of synthesised 
research evidence by policy analysts and advisors in the 
MOHLTC as compared to a 'self-serve' evidence service. 
The full-serve evidence service comprises database access 
(an effort to facilitate policy makers' efforts to 'pull' in 
research when they need it), monthly email alerts about 
new additions to the database (a 'push' effort), and full- 
text article availability (an additional effort to facilitate 
pull). A systematic review found that simply providing 
information (in the form of clinical-practice guidelines) 
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can change clinical behaviour,[ll] which leaves us reason- 
ably confident that we have the potential to achieve an 
increase in evidence use among health-system policy 
mal<ers. Moreover, the results of a cluster randomised trial 
indicate that a full-serve evidence service increased practi- 
cing clinicians' utilisation of evidence-based information 
from a digital library [12]. 

Methods/design 

We will conduct this trial using a sequential explanatory 
mixed-methods design, [13] beginning with the rando- 
mised controlled trial (RCT) and then following up with 
a qualitative process study to explore the RCT findings in 
greater depth. For an initial two-month baseline period, 
all participants will receive the self-serve evidence service. 
For the following six-month period, the intervention 
group will receive the full- serve evidence service and the 
control group will continue to receive the self-serve evi- 
dence service. For a final two-month period, both groups 
will receive the full-serve evidence service. This protocol 
received ethics approval from the Hamilton Health 
Sciences/Faculty of Health Sciences Research Ethics 
Board at McMaster University (project number 10-267). 

RCT methods/design 

Study population and recruitment 

To recruit participants who deal with health-systems 
issues on a regular basis, we will invite all policy analysts 
and policy advisors from one purposively selected divi- 
sion of the MOHLTC to participate in the RCT. All divi- 
sion staff members similarly face a relatively new 
expectation about obtaining training in finding and using 
research evidence (in the form of an indicator in annual 
performance reviews), as well as a new mandate for using 
the Ministry's 'Research Evidence Tool' for submissions 
that support decision making at the Ministry Manage- 
ment Committee and cabinet levels. Moreover, a trial 
endorsement letter will be signed by the Assistant Deputy 
Ministry responsible for this division. These contextual 
developments precede the launch of the trial and help to 
create a favourable climate for the use of research evi- 
dence among all potential trial participants. 

Based on estimates provided to us in June 2009 by the 
MOHLTC, there are approximately 49 policy analysts 
(four are junior program and policy analysts) and 99 
senior policy analysts in the division (n = 148). We do not 
yet have an accurate estimate of the number of policy 
advisors in the division; however, this group is likely to 
include roughly 20 people and all of them are likely to be 
senior policy advisors. By including all three levels of pol- 
icy analysts and (if applicable) both levels of policy advi- 
sors, we will gather evidence from a diverse group that 
plays different roles in the policy-making process. For 
example, a policy analyst might conduct the initial. 



extensive 'workup' of an issue, whereas a senior policy 
advisor might write a short briefing note for the Minister. 

Selecting this sample of policy analysts and advisors 
raises two applicability/generalisabilty issues. First, these 
RCT participants will differ in whether and when they 
received training on finding and using research evidence. 
Two of us (JNL and MGW) delivered a series of five one- 
day workshops for policy analysts and advisors at the 
MOHLTC between July 2008 and March 2009 (i.e., 14 to 
22 months before the trial will begin). We delivered five 
additional one-day workshops, one half-day workshop, 
and one half-day webinar for policy analysts and policy 
advisors, as well as one 1.5-hour workshop for more 
senior MOHLTC executives who set expectations for 
these staff, between January and March 2010 {i.e., two to 
four months before the trial will begin). Given the divi- 
sion's expectation about training, we can assume that 
most RCT participants will have received the training. 
However, newly hired policy analysts and advisors may 
not have received the training, and others may not have 
been able to participate due to scheduling conflicts; those 
that have received the training will differ in the recency 
of the training. Second, these RCT participants will differ 
in their experience, which is somewhat related to their 
position {i.e., level of policy analyst and level of policy 
advisor). To address each of these applicability/generali- 
sability issues, we will stratify the randomisation based 
on past training and current position (see below). 

Intervention and control arms 

We will conduct a two-arm RCT with a full-serve evi- 
dence service as the intervention arm and a self-serve 
version as the control arm. Participants allocated to the 
full-serve evidence service will receive the following: 

• database (Health Systems Evidence) access (facilitat- 
ing pull) 

• monthly email alerts (push) 

• full-text article availability (facilitating pull) 
Participants allocated to the self-serve evidence service 

will receive only database access, which is already pub- 
licly available at http://www.healthsystemsevidence.org. 

Randomisation 

Participants will be randomised using a stratified design. 
After completing the baseline questionnaire during the 
two-month baseline period, participants will be allocated 
to strata based on past workshop attendance (yes or no) 
and their position (policy analyst, senior policy analyst, 
or policy advisor). This two-layer stratification will pro- 
duce six strata. Participants will be randomised after all 
those who consent to participate in the trial have com- 
pleted the baseline questionnaire. We will assign a 
unique participant ID number to each participant and 
then provide the list of IDs to a biostatistician external 
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to the research team who will conduct the randomisa- 
tion and keep a log to provide a clear audit trail. The 
biostatistician will then communicate directly with a 
knowledge broker external to the research team who 
will be generating the email alerts and with the website 
server administrator at McMaster University who will 
be establishing which participants get access to which 
evidence service. The participants and investigators will 
be blinded to group assignment. 

Outcomes 

Measuring the impact of knowledge transfer and 
exchange (KTE) interventions, such as the evidence ser- 
vice proposed here, poses significant challenges [14] . The 
ultimate goal of KTE interventions is typically to improve 
health. However, there is a long chain of potential causal 
relationships between an evidence service and improved 
health. For instance, the evidence service may influence 
the use of research evidence in different stages of the pol- 
icy-making process, which in turn may influence deci- 
sions made by patients and healthcare providers {e.g., 
healthcare professionals, teams, and institutions), which 
may in turn influence whether cost-effective programs, 
services, and drugs get to the patients who need them 
and have their desired impacts, and which in turn may 
translate into improved health [15]. Moreover, even the 
first relationship in this long chain is complicated by the 
competing influences on the policy-making process, such 
as institutional constraints within a political system, sta- 
keholder pressure campaigns, values and beliefs held by 
key decision makers, and external factors such as the 
state of the economy [16-18]. Similar challenges arise 
when assessing the impact of KTE interventions, such as 
guideline-dissemination strategies, on clinical practice 
and on health [19-21]. 

Given these challenges, our primary and secondary 
outcomes for the trial are proxy measures for the use of 
research evidence in policy making. The primary out- 
come will be a measure of utilisation that is similar to 
the one used in a trial of the McMaster Premium Litera- 
ture Updating Service (PLUS) [12]. Specifically, we will 
track the mean number of site visits/month/participant 
across trial groups during each period, that is, the base- 
line period, intervention period, and crossover period. 
We will also provide related descriptive measures such 
as the proportion of users per month in each of the full- 
serve and self-serve groups; the frequency with which 
the full monthly update page, systematic review records, 
and the more detailed documentation for each review 
[e.g., user-friendly summaries, scientific abstracts, and 
full-text reports) are accessed; the mean number of min- 
utes per month that participants use the database (with 
a 'time out' set at 60 minutes); and the number of times 
the monthly email alerts are forwarded. 



Health Systems Evidence will be hosted on a secure 
server at McMaster University and will require a user 
login that will be used to accurately track their usage of 
the database. A user login is necessary because indivi- 
duals from the MOHLTC do not have a consistent IP 
address when accessing external websites, which would 
preclude the collection of utilisation data if the site were 
hosted without requiring users to login. In addition, 
requiring user login will partially protect against con- 
tamination of the control group. However, we cannot 
rule out the possibility that individuals in the interven- 
tion arm of the study will forward monthly email alerts 
and full-text systematic reviews that are available only 
by subscription to individuals in the control arm; how- 
ever, we will collect data about alert forwarding. 

For the secondary outcome, we will use a survey based 
upon the theory of planned behaviour to measure parti- 
cipants' intention to use research evidence. The theory 
of planned behaviour is a model of how human action 
is guided [22,23], and it consists of three variables-atti- 
tudes {i.e., beliefs and judgments), subjective norms {i.e., 
normative beliefs and judgments about those beliefs), 
and perceived behavioural control {i.e., the perceived 
ability to enact the behaviour)-that shape the behaviour 
intentions of people, which is in turn a strong predictor 
of future behaviour [23-25]. In Figure 1, we outline lin- 
kages among the intervention, contextual developments 
(described above), and theory of planned behaviour con- 
structs and measures. 

The theory of planned behaviour has been extensively 
used and tested in the fields of psychology and health- 
care. Systematic reviews conducted in the psychology 
field have demonstrated that the theory explains about 
39% of the variance in intention and about 27% of the 
variance in behaviour [24,25] . A number of studies have 
demonstrated the feasibility of producing valid and reli- 
able measures of key theory of planned behaviour con- 
structs for use with healthcare professionals [26-28]. A 
systematic review suggests that the proportion of the var- 
iance in healthcare professionals' behaviour explained by 
intention was similar in magnitude to that found in the 
broader literature [29]. This successful transfer of the 
theory from individuals (as studied by psychologists) to 
healthcare professionals involved in an agency relation- 
ship with their patients (as studied by health-services 
researchers) bodes well for its further transfer to policy 
analysts and advisors involved in an agency relationship 
with Ministers and other senior officials. 

Using a manual to support health researchers who want 
to construct measures based on the theory [23], we devel- 
oped and sought preliminary feedback on a data collection 
instrument by first assessing face validity through inter- 
views with key informants and then pilot testing it with 28 
policy makers and researchers from 20 low-middle income 
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Elements for both intervention and 
control groups 

• Database access (facilitating pull) 



Intervention element 1 
• Email alerts (push) 



Intervention element 2 

• Full-text article availability 

(facilitating pull) 



Recent contextual developments 

• Expectations about training in 
finding and using evidence 

• Mandating use of tt)e 'Research 
Evidence Tool' for submissions that 
support decision making 

• Trial endorsement letter from senior 
official 



Attitudes 

(behavioural beliefs x outcome evaluations) 
Q4a: Using it is beneficial/harmful 
Q4b: Using it is good/bad 
Q4c: Using it is pleasant/unpleasant 
Q4d: Using it is helpful/unhelpful 



Subjective norms 

(normative beliefs x motivation to comply) 

• Q5; People who are important to me 
thinl< that I should/should not use it 

• Q6: it is expected of me that I use it 
u (agree/disagree) 

Q7: 1 feet under social pressure to use it 
(agree/disagree) 

Q8: People who are important to me 
want me to use it (agree/disagree) 



Perceived behavioural control 
(control beliefs x influence of control/beliefs) 
• Q9: 1 am confident I could use it 
\ (agree/disagree) 
^ QIO: For me to use it is easy/difficult 
Qll: The decision to use it is beyond my 
control (agree/disagree} 
Q12: Whether or not I use it is entirely up 
to me (agree/disagree) 



Behavioural intentions 

• Ql: I expect to use it 

• Q2; / want to use it 

• Q3: 1 intend to use it 



Figure 1 Linkages among tlie intervention, contextual developments, and theory of planned behaviour constructs 



countries who completed it after participating in a KTE 
intervention [30]. In addition, Boyko et al. (2010) found 
moderate test-retest reliability of the instrument using 
generalisability theory (G = 0.50) [31] when scores from a 
sample of 37 health-system policy makers, managers, pro- 
fessionals, citizens/consumers, and researchers participat- 
ing in stakeholder dialogues convened by the McMaster 
Health Forum were generalised across a single administra- 
tion, and even stronger reliability (G = 0.9) when scores 
were generalised across the average of two administrations 
of the tool [30] . In the reliability assessment by Boyko et 
al. (2010), the first administration of the tool immediately 
followed a McMaster Health Forum stakeholder dialogue, 
which may have promoted enthusiasm for using research 
evidence among participants. This likely produced higher 
measures of intention on the first administration of the 
tool as compared to the second, resulting in the lower G 
score. Given that we won't be administering the tool in a 
similar atmosphere of enthusiasm for using research evi- 
dence, we are confident in the level of reliability of the 
tool without two administrations at both baseline and fol- 
low-up. We modified the instrument by adding a question 
to measure the perceived usefulness of the intervention, as 
well as questions about participant characteristics. 



We will administer the instrument during the baseline 
period, as well as at the end of the six-month intervention 
period, through a brief online survey that takes approxi- 
mately 10 minutes to complete. We will use unique identi- 
fiers for each participant to ensure their responses to the 
previous survey are linked for calculations of before-and- 
after changes in their intention to use research evidence. 
We will follow up with participants who do not complete 
the survey once per week for three weeks to minimise the 
number of participants lost to follow-up. 

Data management and analysis 

Data will be entered into SPSS 16.0 (IBM Corporation, 
Somers, NY) after all data collection has been completed. 
Analyses will be conducted by two members of the research 
team (SH and MGW), and during the analysis, neither they 
nor other study investigators will have access to the key 
linking the participants to their unique identifiers. 

We will treat both outcome measures as continuous 
variables and analyse the change in these measures over 
time using a two-way mixed-effects linear repeated-mea- 
sures analysis of variance (ANOVA), with the interaction 
of intervention by time as the main feature of interest. In 
addition, we will control for four variables-past workshop 
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attendance, position (policy analyst, senior policy analyst, 
or senior policy advisor), branch within the division (of 
which there are six), and number of years working at the 
MOHLTC-using analysis of covariance. Given the likeli- 
hood that the distribution of the outcomes will be skewed, 
we will transform the data where necessary and possible, 
which may include adjusting the time period for which we 
calculate the mean number of site visits/participant {e.g., 
calculating the mean over two months) if there are insuffi- 
cient data for analysis. Moreover, as part of a secondary 
analysis, we will assess whether there is an interaction 
between each of these variables (entered as a fixed factor) 
and the outcome measures. We will also qualitatively 
compare the number of participants in the intervention 
and control groups that do not complete the follow-up 
survey and assess whether their baseline characteristics 
can help to explain their loss to follow-up. 

For all analyses, we will use the intention-to-treat 
principle and report 95% confidence intervals; p values 
equal to or less than .05 (two-tailed) will be considered 
significant. For the primary outcome measure (mean 
number of site visits/month/participant), missing data 
are irrelevant because it is a naturalistic measure. For 
the secondary outcome measure (obtained through the 
survey), missing data can be taken into account through 
the use of a mixed-effects model. 

Statistical precision 

Given a fixed sample size of at least 148 policy analysts 
and advisors in the division, a sample- size calculation is 
not relevant. Instead, we have calculated the level of sta- 
tistical precision that we can expect given our fixed 
sample size. We had no mechanism to estimate the 
intraclass correlation coefficient (ICC) for measurements 
of the primary outcome for individuals over time. 
Therefore, we calculated estimates of statistical precision 
for ICCs of .2, .3, .5, .7, and .8 based on a six-month 
trial period with 80% power; an estimated standard 
deviation of 1.0; significance of .05; and 74 participants 
per study group (total n = 148, which does not include 
the as yet undefined number of senior policy advisors). 
Assuming the primary outcome data will be collected 
from all 148 participants at baseline and at six follow-up 
points (one per month), the time-averaged detectable 
difference (in standard deviation units) between the two 
groups is at best 0.27 (ICC = .2), which increases with 
successively greater ICCs to 0.30 (ICC = .3), 0.35 (ICC = 
.5), 0.40 (ICC = .7), and 0.42 (ICC = .8). 

Qualitative study methods/design 

Given that this is the first RCT evaluating a KTE inter- 
vention for health-system policy makers (at least to our 
knowledge) and given the inherent limitations associated 
with measuring research use as an outcome, we will 



conduct a qualitative process study after the completion 
of the trial to explore the RCT findings in greater depth. 
The qualitative study will explore how and why the evi- 
dence service worked (or didn't work), including the 
role of past workshop attendance and position and the 
degree of contamination between the intervention and 
control groups. 

Sample 

We will use a mixed-method sequential nested sampling 
procedure, whereby a larger sample is analysed in one 
study (RCT) and a subset of the larger sample is 
selected for further inquiry in the second study [32]. 
Specifically, 15 participants from each trial arm (n = 30) 
will be purposively sampled [33,34]. Our sampling cri- 
teria include RCT arm {i.e., full-serve or self-serve evi- 
dence service), outcomes, past workshop attendance, 
position, branch within the division, and number of 
years working at the MOHLTC. We have assumed a 
70% response rate (in keeping with our past experience 
with conducting qualitative studies involving health-sys- 
tem policy makers), which means that we should sample 
approximately 40 policy analysts and advisors in order 
to achieve a sample size of 30. 

Data collection 

One-on-one semistructured interviews will be conducted 
either by telephone or in person (where possible) on 
participants' views about and experiences with the evi- 
dence service, including whether and how they used it 
(and the degree of 'contamination' between the two 
arms of the RCT, if any) and why, whether and how it 
was helpful in their work and why, what aspects were 
most and least helpful and why, and recommendations 
for next steps. Potential explanatory factors (for which 
we will probe) include past workshop attendance, posi- 
tion, branch within the division, and number of years 
working at the MOHLTC. 

Data management and analysis 

We will tape and transcribe all interviews, use NVivo 8 
(QSR International, Cambridge, MA) for data manage- 
ment, and use a constant comparative method for analy- 
sis [35-37]. Specifically, two reviewers will identify 
themes emerging from each successive wave of four to 
five interviews and iteratively refine the interview guide 
and emerging themes until we reach data saturation. 
This strategy will allow the reviewers to develop and 
refine codes and broader themes in NVivo 8 that reflect 
the emerging and increasing levels of nuance that result 
from the continuous checks that are involved in the con- 
stant comparative method [35,37]. The same reviewers 
will then apply the final analytic framework to all of the 
interview transcripts and conduct member checking once 
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analysis is completed {i.e., we will send a brief, structured 
summary of what we learned from the interviews and 
invite comment on it). 

Discussion 

To our knowledge, this will be the first RCT to evaluate 
the effects of an evidence service specifically designed to 
support health-system policy makers in finding and using 
research evidence. While there have been a number of 
strategies developed to both support the production of 
policy-relevant research evidence and the identification 
and use of research evidence by health-system policy 
makers [1,38], rigorous evaluations of the effects of these 
strategies remains a critical gap in the KTE literature 
[38,39] . This study will begin to address this gap by pro- 
viding a rigorous evaluation of the effects of a KTE inter- 
vention for policy makers and by examining how and 
why the intervention succeeds or fails. In addition, this 
trial will contribute to an emerging evidence base about 
similarities and differences in 'what works' in KTE across 
different target audiences [6,12,40]. 

The main potential limitation of the RCT is that it will 
be conducted within one division of the MOHLTC, and 
hence, there is the potential for contamination of study 
groups despite the use of a user-specific login. Given 
that many of the policy analysts and advisors work col- 
laboratively, resources from the full-serve evidence ser- 
vice may be shared with those who had been allocated 
to the self-serve arm. Unfortunately, there is no 
mechanism to protect against this fully. However, we 
will adjust for variables (such as the branch in which 
the policy analyst is based) that may be correlated with 
degree of collaboration, and hence likelihood of contam- 
ination; we will measure the number of times that 
monthly email alerts are forwarded; and we will ask 
about contamination in the qualitative process study. 
Furthermore, if we find a significant amount of contami- 
nation through the qualitative study, it suggests that the 
full-serve evidence service is perceived as highly useful 
by those not allocated to receive it. 
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